{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# 9.Cookies专题：MockLogin 模拟登录——以爬取gitHub为例\n",
    "\n",
    "很多情况下，页面的某些信息需要登录才可以查看\n",
    "\n",
    "对于爬虫来说，需要爬取的信息如果需要登\n",
    "才可以看到的话，那么我们就需要做一些模拟登录的事情\n",
    "\n",
    "在前面我们了解了会话和 Cookies 的用法。简单来说，这实际上是在客户端生成了Cookies，而 Cookies 里面保存了SessionID。\n",
    "登录之后的后续请求都会携带生成后\n",
    "的 Cookies 发送给服务器，后者就会根据 Cookies 判断出对应的 SessionID 、\n",
    "进而找到会话。如果会话是有效的那么服务器就判断用户当前已经登录了.\n",
    "返回请求的页面信息.这样我们就可以到登录之后的页面。\n",
    "\n",
    "这里的核心就是获取登录之后的 Cookies 而要获取 Cookies ，我们可以手动在浏览器里输入用户密\n",
    "码，然后再把 Cookies 复制下来，但是这样做明显会增加人工工作量。\n",
    "\n",
    "因此我们将介绍模拟登录的相关方法以及如何维护一个 Cookies池\n",
    "\n",
    "## 9.1 项目介绍与目标\n",
    "\n",
    "如果希望爬取 GitHub 所关注人的最近动态，我们就需要模拟登录GitHub\n",
    "\n",
    "### Part 1 分析登录过程\n",
    "\n",
    "首先要分析登录的过程，需要探究后台的登录请求是怎样发送的，登录之后又有怎样的处理过程\n",
    "\n",
    "如果已经登录 GitHub，先退出同时清除Cookies\n",
    "\n",
    "首先，打开GitHub 登录页面，链接为\n",
    "[https://github.com/login](https://github.com/login)，\n",
    "输入GitHub的用户名和密码，打开\n",
    "开发者工具将 Preserve Log选上，这表示显示持续日志，\n",
    "\n",
    "效果如图所示：\n",
    "![](preservelog.jpg)\n",
    "\n",
    "点击登录按钮，这时便会看到开发者工具下方显示了各个请求过程,找到session，查看headers\n",
    "\n",
    "![](gitsession.jpg)\n",
    "\n",
    "Headers 里面包含了 Cookies Host Origin Referer User-Agent 等信息\n",
    "\n",
    "Form Data 包含了字段， commit 是固定的字符串 Sign in, utf8 是一个句选字符，\n",
    "authenticity_token 较长，其初步判断是\n",
    "Base64 加密的字符串，login 是登录的用户名，password 是登录的密码\n",
    "\n",
    "综上所述，我们现在无法直接构造的内容有 Cookies 和 authenticity_token 下面我们再来探寻\n",
    "下这两部分内容如何获取\n",
    "\n",
    "登录之前我们会访问到一个登录页面，此页面是通过 GET 形式访问的。\n",
    "\n",
    "输入用户名密码，点击登录按钮，浏览器发送这两部分信息，\n",
    "也就是说 Cookies和authenticity_token 是在访问登录页时候设置的\n",
    "\n",
    "时再退出登录，回到登录页，同时清空Cookies，重新访问登录页，截获发生的请求\n",
    "\n",
    "访问登录页面的请求如困所示， Response Headers Set-Cookie字段\n",
    "这就是设置 Cookie 的过程\n",
    "\n",
    "另外，我们发现 Response Headers 没有和 authenticity_token 关的信息，所以可能 authenticity_token\n",
    "还隐藏在其他的地方或者是计算出来的 我们再从网页的源码探寻，搜索相关字段，\n",
    "发现源代码里面隐藏着此信息，它是一个隐藏式表单元素"
   ]
  },
  {
   "cell_type": "raw",
   "source": [
    "<input type=\"text\" name=\"login\" id=\"login_field\" class=\"form-control input-block\" tabindex=\"1\" autocapitalize=\"off\" autocorrect=\"off\" autocomplete=\"username\" autofocus=\"autofocus\">\n",
    "<input type=\"password\" name=\"password\" id=\"password\" class=\"form-control form-control input-block\" tabindex=\"2\" autocomplete=\"current-password\">"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% raw\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "现在我们已经获取到所有信息，接下来实现模拟登录\n",
    "\n",
    "## 9.2 实战\n",
    "\n",
    "### 1> 定义 Login 类，初始一些变量"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import requests\n",
    "from lxml import etree\n",
    "class Login(object):\n",
    "    def __init__(self):\n",
    "        self.headers = {\n",
    "            'Referer':'https://www.github.com/',\n",
    "            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4173.2 Safari/537.36',\n",
    "            'Host':'github.com'\n",
    "        }\n",
    "        self.login_url = 'https://github.com/login'\n",
    "        self.post_url = 'https://github.com/session'\n",
    "        self.logined_url = 'https://github.com/settings/profile'\n",
    "        self.session = requests.Session()\n",
    "# part 2\n",
    "    def token(self):\n",
    "        response = self.session.get(self.login_url,headers=self.headers)\n",
    "        seletor = etree.HTML(response.text)\n",
    "        token = seletor.xpath('//div/input[2]/@value')[0]\n",
    "        return token\n",
    "# part 3\n",
    "    def login(self,usrname,password):\n",
    "        post_data={\n",
    "            'commit':'Sign in',\n",
    "            'authenticity_token':self.token(),\n",
    "            'login':usrname,\n",
    "            'password':password\n",
    "        }\n",
    "\n",
    "        response = self.session.post(self.post_url,data=post_data,headers=self.headers)\n",
    "        if response.status_code ==200:\n",
    "            self.dynamics(response.text)\n",
    "\n",
    "        reslogin = self.session.get(self.logined_url,headers=self.headers)\n",
    "        if reslogin.status_code ==200:\n",
    "            self.profile(response.text)\n",
    "# part 4\n",
    "    def dynamics(self,html):\n",
    "        \"\"\"实现关注人的动态信息爬取\"\"\"\n",
    "        see = etree.HTML(html)\n",
    "        dynamic = see.xpath('//div[@contains(@class,\"news\")]//div[contains(@class,\"alert\")]')\n",
    "        for item in dynamic:\n",
    "            dyn = \" \".join(item.xpath('.//div[@class=\"title\"]//text()')).strip()\n",
    "            print(dyn)\n",
    "    def profile(self,html):\n",
    "        \"\"\"功能：实现处理个人详情页面的信息\"\"\"\n",
    "        see = etree.HTML(html)\n",
    "        dynamic = see.xpath('//input[@id=\"user_profile_name\"]/@value')[0]\n",
    "        ema = see.xpath('//select[@id=\"user_profile_email\"]/option[@value!=\"\"]/text()')\n",
    "        print(dynamic,ema)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    login = Login()\n",
    "    login.login(usrname=\"username\",password=\"password\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "这里最重要的一个变量就是requests库的Session，\n",
    "它可以帮助我们维持一个会话，而且可以自动处理Cookies.我们不用再去担心Cookies的问题\n",
    "\n",
    "## Part 2 介绍\n",
    "接下来，访问登录页面要完成两件事:一是通过此页面获取初始的Cookies，\n",
    "二是提取出authenticity_token\n",
    "\n",
    "在这里我们实现一个token()方法\n",
    "\n",
    "例子：authenticity_token元素位置与内容"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "raw",
   "source": [
    "<input name=\"authenticity_token\" type=\"hidden\" value=\"V+oIrj7RTUp00ODC3JVqIFyCXzUr3/iAetPxZq6wjdVFqG0UXOdlTRBAw+H9d/xkPJ29INpDK9Ta6MfsVzvaNw==\" data-csrf=\"true\">"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% raw\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Part 3 login()的实现\n",
    "\n",
    "现在已经获取初始的Cookies和authenticity_token，开始模拟登录，实现一个login()方法\n",
    "\n",
    "首先构造一个表单，复制各个字段，其中username和password是以变量的形式传递,然后再用 Session\n",
    "对象的post()方法模拟登录即可\n",
    "\n",
    "由于 requests 自动处理了重定向信息，我们登录成功后就可以直接\n",
    "跳转到首页，首页会显示所关注人的动态信息，得到响应之后我们用dynamics()方法来对其进行处理\n",
    "接下来再用Session对象请求个人详情页，然后用profile()方法来处理个人详情页的信息\n",
    "\n",
    "## part 4 dynamics()和profile()方法实现\n",
    "\n",
    "# Cookies 池的搭建\n",
    "\n",
    "很多时候，在没有登录的情况下,我们也可以访问一部分页面或请求一些接口，因为毕竟网站本身需要做 SEO，不会对所有页面都设置登录限制。\n",
    "\n",
    "**但是,不登录直接爬取会有一些弊端，弊端主要有以下两点。**\n",
    "\n",
    "1.设置了登录限制的页面无法爬取。如某论坛设置了登录才可查看资源，某博客设置了登录才\n",
    "可查看全文等，这些页面都需要登录账号才可以查看和获取。\n",
    "\n",
    "2.一些页面和接口虽然可以直接请求，但是请求一旦频繁，访问就容易被限制或者IP直接被\n",
    "封，但是登录之后就不会出现这样的问题，因此登录之后被反爬的可能性更低。\n",
    "\n",
    "我们可以尝试登录之后再做爬取，被封禁的几率会小很多，但是也不能完全排除被封禁的风险。\n",
    "如果一直用同一个账号频繁请求，那就有可能遇到请求过于频繁而封号的问题。\n",
    "\n",
    "如果需要做大规模抓取，我们就需要拥有很多账号，每次请求随机选取一个账号，这样就降低了\n",
    "单个账号的访问频率，被封的概率又会大大降低。\n",
    "\n",
    "那么如何维护多个账号的登录信息呢? 这时就需要用到 Cookies 池了。\n",
    "\n",
    "## 项目实战： Cookies的搭建——以新浪微博为例\n",
    "\n",
    "基本架构图：\n",
    "\n",
    "![](CookiesPool.jpg)\n",
    "\n",
    "### Cookie 池架构的基本模块分为4块：存储模块、生成模块、检测模块和接口模块\n",
    "\n",
    "存储模块负责存储每个账号的用户名密码以及每个账号对应的 Cookies 信息，同时还需要提\n",
    "供方法来实现方便的存取操作\n",
    "\n",
    "生成模块负责生成新的Cookies。此模块会从存储模块逐个获取账号的用户名和密码，然后模拟登\n",
    "录目标页面，判断登录成功，就将Cookie返回并交给存储模块存储\n",
    "\n",
    "检测模块需要定时检测数据库中的 Cookie 在这里我们需设置1个检测链接，不同的站点\n",
    "检测链接不同，检测模块会逐个拿取账号对应的 Cookies 去请求链接，如果返回的状态是有\n",
    "效的，那么此Cookies没有失效，否则Cookies失效并移除，接下来等待生成模块重新生成\n",
    "即可\n",
    "\n",
    "接口模块:要用API来提供对外服务的接口,由于可用的Cookies可能有多,我们可以随机返回Cookies的接口，这样保证每个Cookies都有可能被取到Cookies取到的概率就会越小，从而减少被封号的风险\n",
    "\n",
    "### 1. 存储模块\n",
    "\n",
    "存储的内容无非就是账号信息和Cookies信息，由用户名和密码两部分组成，\n",
    "我们可以存成用户名和密码的映射Cookies可以存成JSON字符串\n",
    "\n",
    "我们需要根据账号来生成Cookies。\n",
    "\n",
    "生成的时候我们需要知道哪些账号已经生成了Cookies，哪些没有生成，\n",
    "所以需同时存该Cookies对应的用户名信息，其实也是用户名和Cookies的映射\n",
    "\n",
    "需要使用Redis的Hash实现"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import redis\n",
    "class RedisClient(object):\n",
    "    def __init__(self, type, website, host=REDIS_HOST, port=REDIS_PORT, password=REDIS_PASSWORD):\n",
    "        \"\"\"\n",
    "        初始化Redis连接\n",
    "        :param host: 地址\n",
    "        :param port: 端口\n",
    "        :param password: 密码\n",
    "        \"\"\"\n",
    "        self.db = redis.StrictRedis(host=host, port=port, password=password, decode_responses=True)\n",
    "        self.type = type\n",
    "        self.website = website\n",
    "\n",
    "    def name(self):\n",
    "        \"\"\"\n",
    "        获取Hash的名称\n",
    "        :return: Hash名称\n",
    "        \"\"\"\n",
    "        return \"{type}:{website}\".format(type=self.type, website=self.website)\n",
    "\n",
    "    def set(self, username, value):\n",
    "        \"\"\"\n",
    "        设置键值对\n",
    "        :param username: 用户名\n",
    "        :param value: 密码或Cookies\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        return self.db.hset(self.name(), username, value)\n",
    "\n",
    "    def get(self, username):\n",
    "        \"\"\"\n",
    "        根据键名获取键值\n",
    "        :param username: 用户名\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        return self.db.hget(self.name(), username)\n",
    "\n",
    "    def delete(self, username):\n",
    "        \"\"\"\n",
    "        根据键名删除键值对\n",
    "        :param username: 用户名\n",
    "        :return: 删除结果\n",
    "        \"\"\"\n",
    "        return self.db.hdel(self.name(), username)\n",
    "\n",
    "    def count(self):\n",
    "        \"\"\"\n",
    "        获取数目\n",
    "        :return: 数目\n",
    "        \"\"\"\n",
    "        return self.db.hlen(self.name())\n",
    "\n",
    "    def random(self):\n",
    "        \"\"\"\n",
    "        随机得到键值，用于随机Cookies获取\n",
    "        :return: 随机Cookies\n",
    "        \"\"\"\n",
    "        return random.choice(self.db.hvals(self.name()))\n",
    "\n",
    "    def usernames(self):\n",
    "        \"\"\"\n",
    "        获取所有账户信息\n",
    "        :return: 所有用户名\n",
    "        \"\"\"\n",
    "        return self.db.hkeys(self.name())\n",
    "\n",
    "    def all(self):\n",
    "        \"\"\"\n",
    "        获取所有键值对\n",
    "        :return: 用户名和密码或Cookies的映射表\n",
    "        \"\"\"\n",
    "        return self.db.hgetall(self.name())"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "这里我们新建了一个RedisClient类，初始化＿init__()方法有两个关键参数\n",
    "\n",
    "type和website分别代表类型和站点名称，它们就是用来拼接 Hash 名称的两个字段\n",
    "\n",
    "如果这是存储账户的Hash，那此处的type为accounts,website为weibo ，\n",
    "如果是存储Cookies的Hash那么此处的type为cookies,website为weibo.\n",
    "\n",
    "### 2.生成模块\n",
    "\n",
    "生成模块负责获取各个账号信息并模拟登录，\n",
    "随后生成Cookies保存\n",
    "\n",
    "我们首先获取两个Hash的信息，看看Hash比Cookies的Hash多了哪些还没有生成 Cookies 的账号，\n",
    "然后将剩余的账号遍历，再去生成 Cookies 即可\n",
    "\n",
    "这里主要逻辑就是找出那些还没有对应Cookies的账号，然后再逐个获取Cookies\n",
    "\n",
    "### 3.检测模块\n",
    "\n",
    "生成模块免不了 Cookies 失效的问题，例如时间太长导致\n",
    "Cookies 失效，或者 Cookies 使用太频繁导致无法正常请求网页\n",
    "\n",
    "如果遇到这样的 Cookies，我们肯不能让它继续保存在数据库里，这时需要检测\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "class ValidTester(object):\n",
    "    def __init__(self, website='default'):\n",
    "        self.website = website\n",
    "        self.cookies_db = RedisClient('cookies', self.website)\n",
    "        self.accounts_db = RedisClient('accounts', self.website)\n",
    "\n",
    "    def test(self, username, cookies):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def run(self):\n",
    "        cookies_groups = self.cookies_db.all()\n",
    "        for username, cookies in cookies_groups.items():\n",
    "            self.test(username, cookies)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "在这里定义了一个父类叫作 ValidTester ，在＿init__()方法里指定好站点的名称website，另外\n",
    "建立两个存储模块连接对象 cookies_db和accounts_db，分别负责操作 Cookies和账号的Hash,\n",
    "\n",
    "我们需要写一个子类来重写这test()方法，每个子类负责各自不同网站的检\n",
    "测，如检测微博的就可以定义为 WeiboValidTester ，实现其独有的 test()方法来检测微博的 Cookies\n",
    "是否合法，然后做相应的处理，所以在这里我们还需要再加一个子类来继承."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "class WeiboValidTester(ValidTester):\n",
    "    def __init__(self, website='weibo'):\n",
    "        ValidTester.__init__(self, website)\n",
    "\n",
    "    def test(self, username, cookies):\n",
    "        print('正在测试Cookies', '用户名', username)\n",
    "        try:\n",
    "            cookies = json.loads(cookies)\n",
    "        except TypeError:\n",
    "            print('Cookies不合法', username)\n",
    "            self.cookies_db.delete(username)\n",
    "            print('删除Cookies', username)\n",
    "            return\n",
    "        try:\n",
    "            test_url = TEST_URL_MAP[self.website]\n",
    "            response = requests.get(test_url, cookies=cookies, timeout=5, allow_redirects=False)\n",
    "            if response.status_code == 200:\n",
    "                print('Cookies有效', username)\n",
    "            else:\n",
    "                print(response.status_code, response.headers)\n",
    "                print('Cookies失效', username)\n",
    "                self.cookies_db.delete(username)\n",
    "                print('删除Cookies', username)\n",
    "        except ConnectionError as e:\n",
    "            print('发生异常', e.args)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 4. 接口模块\n",
    "\n",
    "生成模块和检测模块如果定时运行就可以完成Cookies实时检测和更新，但是 Cookies 最终还是\n",
    "需要给爬虫来用，同时一个Cookies池可供多个爬虫使用，所以我们还需要定义一个Web接口，\n",
    "访问此接口便可以取到随机的Cookies\n",
    "\n",
    "我们采用 Flask 来实现接口的搭建，"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from flask import Flask, g\n",
    "__all__ = ['app']\n",
    "\n",
    "app = Flask(__name__)\n",
    "\n",
    "@app.route('/')\n",
    "def index():\n",
    "    return '<h2>Welcome to Cookie Pool System</h2>'\n",
    "\n",
    "\n",
    "def get_conn():\n",
    "    \"\"\"\n",
    "    获取\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    for website in GENERATOR_MAP:\n",
    "        print(website)\n",
    "        if not hasattr(g, website):\n",
    "            setattr(g, website + '_cookies', eval('RedisClient' + '(\"cookies\", \"' + website + '\")'))\n",
    "            setattr(g, website + '_accounts', eval('RedisClient' + '(\"accounts\", \"' + website + '\")'))\n",
    "    return g\n",
    "\n",
    "\n",
    "@app.route('/<website>/random')\n",
    "def random(website):\n",
    "    \"\"\"\n",
    "    获取随机的Cookie, 访问地址如 /weibo/random\n",
    "    :return: 随机Cookie\n",
    "    \"\"\"\n",
    "    g = get_conn()\n",
    "    cookies = getattr(g, website + '_cookies').random()\n",
    "    return cookies\n",
    "\n",
    "\n",
    "@app.route('/<website>/add/<username>/<password>')\n",
    "def add(website, username, password):\n",
    "    \"\"\"\n",
    "    添加用户, 访问地址如 /weibo/add/user/password\n",
    "    :param website: 站点\n",
    "    :param username: 用户名\n",
    "    :param password: 密码\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    g = get_conn()\n",
    "    print(username, password)\n",
    "    getattr(g, website + '_accounts').set(username, password)\n",
    "    return json.dumps({'status': '1'})\n",
    "\n",
    "\n",
    "@app.route('/<website>/count')\n",
    "def count(website):\n",
    "    \"\"\"\n",
    "    获取Cookies总数\n",
    "    \"\"\"\n",
    "    g = get_conn()\n",
    "    count = getattr(g, website + '_cookies').count()\n",
    "    return json.dumps({'status': '1', 'count': count})"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PyCharm (PycharmWorkspace)",
   "language": "python",
   "name": "pycharm-7b63a738"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}